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Abstract 

This invited paper briefly describes our progress in developing a 
resilient multi-spectral image analysis capability for remote sensing 
applications. This capability is intended to allow image streams from a 
collection of distributed sensors to be disseminated and interpreted by 
a group of analysts, while under information warfare attack. There are 
five component technologies that we are developing: real-time multi- 
and hyper-spectral camera systems, concurrent algorithms for image 
analysis, high performance networking and computer architectures, 
algorithms for achieving computational resiliency, and general 
mathematical tools for integrating these technologies. 


1. Introduction 

The end of the cold war signals a radical change in the nature of threats that the United States must now 
confront. Significant among these threats is the increasing capability of radical regimes abroad and the 
emergence of international terrorism at home. The key to security in this environment is information. 
High-quality surveillance and analysis techniques are at a premium, yet surveillance alone is not enough: 
technologies must be developed that allow sensor information to be fused and disseminated, in real-time, 
to multiple observers and controllers while under information warfare attack. These technologies must 
provide a collaborative problem-solving medium with the ability to sense, interpret, analyze, and decide 
upon dynamically unfolding situations. Our research is concerned with the architectural issues associated 
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with this general research direction. Toward this end we are developing, integrating and evaluating five 
technologies: real-time multi- and hyper-spectral camera systems, concurrent algorithms for image 
analysis, high performance networking and computer architectures, algorithms for achieving 
computational resiliency, and general mathematical tools for integrating these technologies. The sections 
that follow briefly review our progress to date in each of these areas. 


2. Real-time Camera System 

In a joint collaboration with Mobium Enterprises Inc. and Integrated Scientific Imaging we have 
developed a commercially available real-time, multi-spectral camera system (Figure 1). The system is 
capable of acquiring and analyzing real-time multi-spectral imagery using concurrent architectures. It 
incorporates a Kodak Megaplus ESl.O camera capable of delivering 12 bit/pixel images at 60 frames/sec 
(EPS) at the full resolution of 1024x1024 pixels, or 120 EPS in a 2x2 binning mode at 512x512 pixels. A 
motorized filter wheel and interchangeable optical system allow 12 independent spectral bands, from 400- 
lOOOnm, to be delivered. Under Air Force contracts, we are exploring designer lenses to fit this system 
that can be built to suite the optical needs of specific applications. 



Figure 1. Multi-spectral Camera System 

The camera system is interfaced directly into the dual-PCI bus of commercial off-the-shelf (COTS) 
shared-memory multi-processors; these in turn can be connected through gigabit networking for 
scalability. The system has been operational for several months and is currently being integrated with the 
other technologies described in this paper. 


3. Concurrent Algorithms 

We are investigating two primary concurrent algorithms based on the principal component and 
independent component transforms [Lee 1998]. In collaboration with Mobium Enterprises we have 
developed a concurrent spectral-screening PCT algorithm that can be used for hyper-spectral image 
fusion in remote sensing applications [Achalakul 1999, 2000a, 2000b]. The algorithm combines the 
Principal Component Transform (PCT) [Mackiewicz 1993, Singh 1993] with spectral angle classification 







[Kruse 1993] and human-centered color mapping [Boynton 1979, Peterson 1993, Poirson 1993]. It can be 
used on networks of multi-processors consistently with the camera system described in Section 2. 


The PCX is generally used to summarize and de-correlate images by removing redundancy and packing 
the residual information into a small set of images, termed principal components. To prevent the PCX 
from highlighting only the variation that dominates numerically, we augment it with spectral angle 
classification prior to the de-correlation process. This has the effect of reducing the importance of an 
object that occurs frequently in a scene. For example, the spectral signature of a mechanized vehicle 
embedded in a forest scene will be treated as equally important as the signature associated with trees. The 
final step of the algorithm is to generate a color-composite image from a collection of principal 
components. To achieve this, we use a human-centered approach that attempts to match the spatial- 
spectral content of the output image with the spatial-spectral processing capabilities of the human visual 
system. 

The concurrent algorithm uses the standard manager/worker decomposition technique [Chandy 1992]. 
The manager thread partitions the problem and distributes the sub-problems to worker threads. The 
workers solve each allocated sub-problem, send back a result, and wait for the next sub-problem. Each 
sub-problem is a sub-cube of the hyper-spectral image set similar to the decompositions used in [Palmer 
1998]. To reduce communication overhead, a worker overlaps the request for its next sub-problem with 
the calculation associated with the current sub-problem. 

The algorithm was tested using a 210-channel hyper-spectral image collected with the Hyper-spectral 
Digital Imagery Collection Experiment (HYDICE) sensor, an airborne imaging spectrometer. These 
images correspond to foliated scenes taken from an altitude of 2000 to 7500 meters at wavelengths 
between 400nm and 2.5 micron. The scenes contain mechanized vehicles sitting in open fields as well as 
under camouflage. Eigure 2 shows two frames picked from the 210 spectral bands. 



Figure 2: 400 and 1998 nm 

Eigure 3 shows the resulting image after applying the concurrent spectral-screening PCX to the full 210- 
frame data set. The color-mapping scheme maps the first principal component to achromatic, the second 
to red-green opponency, and the third to blue-yellow opponency. The result, when viewed on a high- 
quality monitor, shows significantly improved contrast levels. The forested areas show significantly 
improved detail and the camouflaged vehicle in the lower left corner is significantly enhanced against its 
background. 







Figure 3: Color-Composite Image 

The performance of the algorithm was measured on an 8-way 550MHz Intel shared memory multi¬ 
processor. Using this technology it is capable of processing one 210-band image cube in 10 seconds and 
12 bands in 1/3 second [Achalakul 2000b]. The algorithm has also been tested on a distributed network 
of 16 Sun Solaris 300MHz.workstations connected with 100BaseT networking technology [Achalakul 
2000a]. The performance improvement was linear with the number of processors to within 5% on multi¬ 
processors and 20% on networked workstations. We are currently beginning evaluations of a gigabit 
network of multi-processors and expect this to be completed by the end of the summer. All of the 
concurrent aspects of the spectral-screening PCT algorithm carry over directly to the independent 
component transform that is currently under investigation. 


4. Networking and Architectures 

In addition to off-the-shelf networking and multi-processors technologies, we are also exploring the use 
of special purpose image processing architectures. Our current work is focused on the BMDO 
VIGILANTE multi-spectral sensing technology [Duong 1997, Padgett 1997]. 

The heart of this project involves the development of the ANTE processor — a sugarcube sized 3D VLSI 
chip stack, which performs high-speed convolutions. The ANTE processor uses a neural circuit design 
based on Multiplying Digital-to-Analog Converter (MDAC) technology. The architecture allows 64 
complete inner products, each with a 4096 (i.e., 64x64) input array to be accomplished in 250 
nanoseconds (i.e., 10^^ multiply and add operations in 1 second). We have worked with the Jet 
Propulsion Laboratory to ensure that the technology can be integrated directly into the dual-buss PCI slots 
of a COTS multi-processor. This allows the device to be integrated with all of the other component 
technologies described in this paper. The first prototypes of the sugarcube are expected to be available in 
the summer of 2000. 



5. Computational Resiliency 


Any system that operates in highly adverse environments, such as battlefield command and control, must 
be able to tolerate information warfare attacks and networking/processor failures. The Air Force strategy 
to provide operational survivability is based on the notion of information resiliency: the ability of a 
system to tolerate, dynamically reconfigure, and repair itself in response to an attack [Giordano 2000]. 
This strategy incorporates data resiliency for databases and sequential systems, computational resiliency 
for distributed applications, and real-time attack assessment for Cyberforensics. 

To understand how computational resiliency operates, consider a distributed application as analogous to 
an apartment complex inhabited by a new strain of roach (process/thread). The roaches are highly 
resilient: you can stamp on them, spray them, strike them with a broom but you never kill them all or 
prevent them from their goal of finding food (resources). To foil your eradication efforts, they use several 
techniques: they are highly mobile moving from one place in the apartment complex (network) to another 
with speed and agility. They continually replicate to ensure that it is not possible to kill them all. They 
sense (attack assessment) their environment to obtain clues that mobility is necessary: if a light is turned 
on, they scurry away in all directions to hide behind cupboards in places of known safety (secure network 
zones). If a new roach killer is invented they learn from it, and adapt their behavior to compensate. 
However, this new strain is particularly aggressive and seeks to live in the daylight (wide-area operation): 
thus it adopts techniques for camouflage as a form of protection and disinformation.^ 

There are several significant technical challenges involved in developing systems based on this notion. 
Techniques must be developed for providing policy-driven on-the-fly replication, camouflage, and mobile 
threads. There are also a number of serious theoretic concerns that relate to race conditions in 
reconfiguration of a distributed application, resource management, and providing guarantees on message 
delivery. The techniques must operate on heterogeneous clustered environments that include shared 
memory multi-processors, high performance networking, and wireless networking. These systems are 
composed of machines with substantively different memory and processor characteristics, operating 
systems, floating point representations, and byte orderings. 

To date we have designed and implemented a prototype mobile thread capability that incorporates 
dynamic replication and the associated communication protocols. Figure 4 illustrates how this technology 
is used to organize a distributed computation. Driven by mission policies, critical threads are chosen for 
replication. In the event that one of the replicated threads is compromised, the remaining replicas are used 
to dynamically regenerate a new replica, at an alternative location in the network. Communication is 
automatically reconfigured to the new thread. 
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Thanks to Cathy McCollum for providing the analogy. 



This approach assures operational survivability to the required level of redundancy, subject only to the 
constraints imposed by the total available resources. Obviously to be successful, the replacement thread 
must be mapped to an alternative location in the network with sufficient resources. 

The mobile thread capability has been applied to three applications: a simple fluid dynamics problem 
based on heat diffusion, a target tracking problem using sonar, and image fusion using the spectral- 
screening PCT described in a previous section [Achalakul 2000a]. In this latter experiment, the 
computational overhead associated with the protocols for resiliency was less than 10%, plus the 
unavoidable cost of replication. The protocols have subsequently been simplified, however, new 
performance data is not yet available. 


6 . Mathematical Tools 

In collaboration with Mobium Enterprises Inc., we have developed a commercial tool, Mathweb, that 
provides concurrent image manipulation and linear algebra [Achalakul 1999]. It operates on a single PC, 
Unix machine, or shared-memory multi-processor. A version for distributed systems, based on web- 
technologies, is currently under development; This tool provides a collaborative environment for 
distributed dissemination and analysis of sensor data. 

MathWeb generalizes the matrix algebra familiar to users of Matlab, IDL, and Mathematica to full tensor 
algebra. To understand the relevance of this concept to multi- and hyper-spectral image analysis, consider 
the hierarchy in Figure 5. Using the alternative interpretations, matrix algebra can be generalized to the 
manipulation of multi- or hyper-spectral image streams. The resulting algebra of images allows these 
streams to be manipulated using standard well-defined linear algebraic concepts such as matrix inversion, 
EFT, wavelet transformation, digital filtering, and principal/independent component transformation. 
MathWeb provides a large variety of operations on tensors that can be used collaboratively through web 
technologies to operate on real-time multi-spectral image streams. 


Tensor Dimension 

Conventional Interpretation 

Alternative Interpretation 

0 

Scalar 

Real or Complex Number 

1 

Vector 

List of Values 

2 

Matrix 

Table of values or Image 

3 

Tensor 

RGB-Image 

N indexed by wave length 

Tensor 

Multi-and Hyper-spectral Image 

N indexed by time 

Tensor 

Video Stream 


Figure 5. Tensor Hierarchy 


7. Conclusion 

This paper has briefly described our research in exploring an architectural framework for the distributed 
dissemination and analysis of real-time sensor data. The architecture is intended to allow the fusion of 
sensor data originating from multi- and hyper-spectral cameras, radar and sonar devices, and other sensors 
used in the battlefield environment. The intent is to assure operational survivability of such applications 




from information warfare attack through resiliency concepts. To date we have focused upon multi- and 
hyper-spectral imagery as a vehicle to explore these ideas, by combining and adapting well-known 
approaches to the distributed environment. 
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